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ABSTRACT 



A checklist of characteristics of effective mail 



questionnaires was developed that re-^'lects some degree of consensus 
aunong experts in survey research anu that can serve as a guide to 
novices designing a que'itionnaire. Journal and book sources of 
information about mail questionnaires were reviewed to yield 83 items 
in 7 categories that comprised the study survey. The survey was 
mailed to six authors of books on survey research and six experienced 
practitioners of survey research. Only one author failed to reply; 
the response rate was 92%. Detailed background information was 
provided by 10 of the 11 individuals who participated in this phase 
of the study. iSight of the 83 items were judged desirable for all 
mail questionnaires, possibly because of a lack of clarity in the 
instrument. A revised instrument was submitted to a validation panel 
of 10 individuals experienced in survey research as well as to 8 of 
the original 11 participants. Of the 64 items that a majority of the 
participants in the first phase of the study would usually recommend, 
38 were supported by the validation panel, with 80% or higher 
indicating that they also would usually make the recommendation. The 
attached "Check List of Desirable Characteristics of Mail Surveys" ir> 
a compilation of the items usually recommended by at least 87.5% of 
the first group of experts and 80% of the validation panel. These 
results indicate that questionnaire <^*esign may be a science up to a 
certain point, but beyond that point it is an art. A bibliography of 
the 21 journal and book sources that were reviewed is provided. The 
survey instrument used in this study is included. (SLD) 
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IMPROVING THE QUALITY OF QUESTIONNAIRES: 
A TOOL FOR EVALUATORS 

Background and Objective 

Mail surveys are used frequendy, particulariy in higher education institutions (Fuqua, 
Hartman, & Brown, 1982). According to Babbie (1973), "survey research is probably die best 

known and most widely used research mediod in die social sciences today To some extent, 

everyone in die United States at least has been affected by surveys" (p. i). While die research 
instrument is only one conponent of die overall research endeavor, in mail surveys the 
questionnaire takes on added importance. The potential respondoit encounters it in isdation, widi 
no interviewer picsent to encourage die respondent to participate or to provide explanations. The 
individual must be motivated to complete the questionnaire, and die questiotuiaire must be designed 
U> facilitate die respondent's providing valid responses. "The task required of respondents must 

appear to be easy and attractive Anything [respondents] particularly dislike about die layout, 

wording, or emphasis of die questions may deter diem (Hoinville, 1978, p. 127)." Berdie, 
Anderson, and Niebuhr (1986) concur, noting diat "poorly constructed formats ((die physical 
arrangement of questions on die page)] influence not only response rates but also die quality of 
responses obtained" (p. 23). 

Not suijHisingly, one of die most commonly used techniques for collecting evaluation data is 
die paper-and-pencil questionnaire. Because the value of die data-and die evaluation itsclf--is at 
least partially dependent on die quality of die data collection device, it is important diat evaluators 
be knowledgeable about principles of questionnaire design. As Riecken (1972, p. 94), points out, 
"die development of measuring devices is a technical problem of social science radier dian one 
peculiar to evlauation studies." The literature on social science survey research is a particularly 
relevant resource when evaluation data are collected by mail. 

The body of literature on survey research includes a number of worics diat describe various 
audiors' systems of total survey design-including the design of die instrument itself, sampling 
considerations, choosing questions, cover letters, follow-up pnxedures, etc. Also in die literature 
is a pleihon of studies dealing widi various effects (e.g., on response rate, on turnaround time) <rf 
alternate forms of one cm- more elements of questionnaire design (e.g., multiple chdce versus open- 
ended responses, variations in type size or type style). One would be haid pressed, however, to 
identify a simple check list of characteristics of effective mail questionnaires diat might provide 
useful guidance for die evaluator. The total system concept is too restrictive for diis purpose: not 
only do many of die suggestions offered represent a single point of view, but die guidelines may 
be so specific diat diey are difRcuk to generalize to a simation odier than diat for which diey are 
illustrated. The problem widi recommendations found in most journal articles is that diey are likely 
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to be too narrow in scope for the purpose described The objective of the present study was to 
develop a check list of characteristics of effective mail questionnaires (a) that reflects some degree 
of consensus of experts in survey research, and (b) that can be used as a general guideline by 
novice questionnaire designers. 

Phase One 

Instrument Development 

Significant, journal and book sources that provide general guidelines for designing mail 
questionnaires were identified through an investigation of current books in print and ERIC listings. 
Those sources containing guidelines only for other survey techniques (e.g., telephone or face-to- 
face interviews) were not included unless it could be determined that the recommended procedures 
werp equally applicable for mail surveys. The final list of sources is attached as the Bibliography. 

The specific recommendations from each of these books or articles were listed and categorized. 
Only those charaaeristics which were deemed desirable by several of the authors were retained; 
those characteristics mentioned by only a few authors and those about which there was 
disagreement were excluded from the list The remaining list of desirable characteristics was edited 
to exclude redundancies and re-categorized independently by the two authws. The authors then 
compared and discussed their respective lists ultimately producing the 83 items and seven 
categories contained in the Phase One instrument Throughout their discussions the authors 
recognized that there was mcwie than one way in which items could be categorized, and that the 
categorization of items and the labels selected were somewhat arbitrary. 

The final grouping of items resulted in seven categories requiring varying numbers of 
responses. The cate^ries and their respective numbers of items (responses) were as follows: 
General Appearance (14 items); Instructions (8 items); Choice of Items (8 items); Order of Items 
(15 items); Item Format (16 items); Choice of Response Options (10 items); and Wording (12 
items). 

Once the characteristics had been selected, it was apparent that some might be more important 
than others. To this end, the respondents were asked to indicate, for tach item, the extent to which 
the characteristic would be recommended for mail survey questionnaires using the following 
ratings: "all" ("recommended for all mail survey questionnaires"); "some" ("recommended for 
some but not all mail surveys"); ot "none" ("not recommended"). There was space following each 
section under a heading of "other" for the respondents to add other characteristics that they thought 
should have been included in that section. 

The questionnaire was photocopied and assembled in booklet format (7" by 8-1/2") using two 
sheets of ivory colored, legal-sized paper that were printed on both sides, folded, collated, and 
saddle-stitched. The front page served as a cover and the last (or eighth) page was reserved for 




comments. Wcntical directions for responding to the items were placed at the top of each of the six 

inside pages. 

Participants 

It was considered inqxxtant that the participants in this study be knowledgeable and 
experienced in survey research and represent various research environments. Six authors or books 
on survey research (from the attached bibliography) were invited to participate. One declined. 
Their publications contained guidelines for the total develq)ment of mail questionnaires (including 
wording, order, and format or layout) and were not specific to a particular research emphasis 
(acadcmia, public (pinion polls, marketing research). Six experienced practitioners of survey 
research were selected from the membership of the American Educational Research Association's 
(AERA's) Special Interest Group on Survey Research in Educauon on the basis of their activities 
in the group and their survey research background. 

Detailed background infomiation was provided by ten of the eleven individuals who 
participated in this phase of the study. In addition to the five who were authors of books on 
survey meUxxlology, each of the remaining six had made formal presentations on issues of survey 
research methodology at national professional conferences. Each of the five authws is in a 
leadership position in an (x-ganization which has a focus on survey research. Four of the other 
participants are employed in postsecondary institutions in units that focus on research and/or 
evaluation. 

The ten who supplied background information had amassed a total of 172 years of experience 
in survey research, with individual experience varying from 7 to 40 years (median = 15.5 years). 
All ten had carried out surveys during the previous year, and roost considered it a typical year. 
These researchers had conducted from 1 to 30 surveys themselves, as well as providing 
consultation on others. The research focus and the target population varied both within and across 
individuals. The majw types of surveys were described as public opinion, needs assessment, 
program evaluation/effectiveness, and institutional, consumer, and attitude studies. Target groups 
enumerated included the foF >wing: the general public; program participants; students; alumni; 
consumers; client groups; various occupational groups, including professionals (e.g., judges, 
lawyers); and groups of employees within organizations (e.g., supervisors, managers). 
EEQCfiduiSS 

A copy of the instrument, an explanatory cover letter, and a pre-stamped reply envelope were 
mailed in the spring of 1988. One follow-up reminder which included another copy of the 
instrument was mailed to each of the nonrespondents approximately one month after the initial 
mailing. Only one of the twelve potential participants in this phase of the study did not respond, 
for a response rate of 92 percent. 
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Results 

Only eight of the 83 items were judged to be desirable for "all" mail questionnaires by all of the 
respondents. Some items lacked clarity, as indicated by the comments of the respondents. There 
were two items that none of the respondents recommended for "all" mail questionnaires. 
Cbmments from the researchers' professional colleagues led them to ccmclude that the poles of the 
initial response categories ("all" and "none") may have been too extreme, causing participants to 
reject them for the only other available option, "some" ("recommended for some but not all mail 
surveys") if they thought of one or more applications for which the recommendation would or 
would not be made. Limitations of the instrunjcnt were thus considered as a possible facu>r 
contributing to the lack of consensus among the experts. Phase Two of the study was then 
undertaken. 

Phase Two 

Instrument Revision 

Concern about effects of the response options led the researchers to revise the instrument. The 
response opnon "all" ("recommended for all mail survey questionnaires") was broadened to 
"usually=usually or always recommended for mail survey questionnaires" while the other 
extreme, originally stated as "none" ("not reconmiendcd"), was revised to become 
"seldom=seldom or never recommended for mail survey questionnaires." The middle category of 
"some" ("recommended for some but not all mail surveys") was changed only to reflea the tone 
and phrasing of the new response options, becoming "sometimes=sometimes recommended for 
mail survey questionnaires." 

Almost all of the items from Phase One w^^re used in the revised questionnaire. Four of the 
five items which had caused confusion to participants in the previous study were rewritten in an 
effon to clarify them, and the fifth such item was deleted. 

The revised instrument contained 82 items in seven categories: General Appearance (14 items). 
Instructions (8 items). Choice of Items (7 items). Choice of Response Options (10 items). 
Wording (12 items). Order of Items (15 items), and Item Format (16 items). 

Two versions of the questionnaire were produced. The items themselves were identical, but in 
one version, following each section of items (a section contained items of one of the seven types), 
participants were asked to indicate the circumstances or types of surveys in which items rated as 
"sometimes" would be recommended. If rfiere was not sufficient blank space at the bottom of the 
page, the facing page was left blank for this purpose. In the other version, there was no space for 
participants to explain or list circumstances relevant to items they had rated as "sometimes." A 
copy of the second-version questionniiire is appended to this paper. 

Questionnaires of the first type were duplicated on blue legal size paper that was stapled in the 
middle to form a booklet. Because of the additional space required for explanatory comments, the 



questionnaire had 12 pages. Quesuonnaires ci the second type were green and required only eight 

pages. 

Participants 

The researchers wanted to take advantage of the extraordinary credentials <^ the 1 1 experts 
surveyed in Phase One, but they were concerned about the potential bias of the results caused by 
that group's prior exposure to the instrument. For this reason, a second group, a "validation 
panel," was identified 

A group size of 10 was deemed desirable for Uiis validation panel. Oversampling was initiated 
by sending questionnaires to 15 individuals to obtain 10 participants for the validation panel. The 
respondents in die validation group were the first 10 individuals (from the 15 who were sent 
questionnaires) who retumed completed questionnaires. In all, a total of 12 from the validation 
sample retumed the survey instruments without benefit of fdlow-ups. Background information 
was provided by nine of the 10 validation panel members. 

For the validation group, experience in survey research activities varied from five to 30 years, 
with a median of 13 years. Participants had been involved in conducting firom none (n=l) to eight 
(n=l) mail surveys in 1988, with a mean of tiiree. Five of the nine participants indicated 1988 was 
typical of their survey activities, but the remaining four indicated they were usually more active in 
survey research. Eight of the nine had conducted and repotted studies of survey or questionnaire 
methodology, and four had published articles or books on survey mediodology. Eight (rf^ the 10 
individuals were involved in institutional ot organizational research, and five of diem limited their 
activities to this type. Otfier types of survey research included public opinion (n=2), consumer 
(n=l) and social science (n=l). Populations surveyed included members of organizations (n=5), 
program participants (n=5), alumni (n=4), and the general public (n=3). Seven of die 10 were 
college/university faculty members, with three employed by research institutes witfiin their 
insututions. Two individuals were enq)loyed by research divisions or sections in large 
organizations, and one individual was in a public schod research division. 

In addition to die validation panel, die 11 participants from Phase One were given an 
<q)portunity for continued involvement in die project Of these 1 1, eight completed questionnaires. 
Two odiers indicated diat diey diought dieir responses would be biased by dieir participation in 
Hiase One and declined to participate. One of die two recommended a colleague to participate in 
his place, and die odier fwwarded his questionnaire to a colleague. Bodi recommended indivkluals 
became pan of die validation group. Only one of die original 1 1 participants failed to respond in 
any manner. 
Procedures 

The 12-page survey forms, cover letters and postage-paid return envelopes (along vyith 
previously promised copies of die results of Phase One) were mailed to die original 1 1 participants 



in the spring of 1989. The cover letter emphasized the change in response options. One follow-up 
mailing, containing a letter, a replacement copy of the questionnaire and a stamped, addressed 
return envelope, was sent approximately one month later, Completed survey instruments were 
received fiom eight of the 11 original participants (73 percent). 

Eight-page survey forms, cover letters and postage-paid return envelopes were mailed to the 15 
individuals selected for the validation sample in the summer of 1 989. Responses from 1 2 of them 
constituted an 80 percent response rate. 
Analysis 

Frequency distributions were prepared for all items for the Phase One participants and for the 
validation panel. Explanatory comments listing special circumstances in which practices would 
scwietimes be recommended were noted for items on which there was not total agreement by 
participants from Phase One. 

Items were listed in four groups, based on responses of the Phase One participants: items on 
which all who responded marked the item as "usually" recommended, items on which all but one 
who responded marked the item as "usually" recommended, items on which all but two who 
responded marked the item as "usually" recommended, and the items that lacked general acceptance 
as usual practices. 

After items were grouped according to responses by the original participants, the percentage of 
validation panel members who would "usually" recommend each item was calculated. If an item 
was "usually" recommended by 80 percent or more of the validation panel, the item was 
considered to have been su pported. 

Phase Two Results and Discussion 

On 26 of the 82 items (32 percent), all eight of the original panicipants agreed that they would 
"usually" recommend the item for mail survey questionnaires. And on another 8 items, all of 
those responding to the item (n=6 or n=7) rated the items as usually recommended. Those items 
are listed below. 

On 25 of the 34 items, the validation panel agreed at the 80 percent level or higher. Those 
items appear in bold-face type in the list that follows. The 9 items that do not appear in bold-face 
type were supported by fewer than 80 percent of the validation panel. 
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Items Recommended by AH of Original Group 



A. GENERAL APPEARANCE 

1. The title of the study/questionnaire is likely to appeal to the survey population. 

2. Instrument looks easy to complete. 
(• Type is clear and legible. 

11. Appreciation for completing the instrument is expressed. 

9a. The Cront page (or cover) contains the study/instrument title, prominently displayed. 

B. INSTRUCTIONS 

2. Instructions are brief. 

9a. Instructiois are clear: They specify when to put a check mark and when to 
write in a response. 

3b. Instructions are clear: They indicate whether multiple responses are allowed. 

6. If items appear on both sides of the page, an indication is given that the instrument continues on the 
reverse side (e.g., "please turn over"). 

C. CHOICE OF ITEMS 

2. Each item seeks just one piece of information. 

3. All items are essential and relevant to the purposes of the survey. 

4b. For items used for skip/niter/screen purposes, instructions are few and simple. 

D. CHOICE OF RESPONSE OPTIONS 

la. Response options exhaust all possibilities or include "other/* "undecided," or 
"neutral" category. 

Id. Response options do not contain more than one alternative that could be correct 

unless multiple responses are allowed. 
Ig. Response options are appropriate for the item. 

1 f . Response options are brief. 

2b. Items with Likot-type response options use a balanced scale. (n=7) 

E. WORDING 

1. The choice of words is appropriate to the literacy level of the survey population. 
3d. Items are simple, direct, and unambiguous. They do nn^ contain instances of 

double negatives in items and/or response options. (ns7) 
3t. Items are simple, direct, and unambiguous. They do not contain instances of 

negatively worded items coupled with agree/disagree response format. (n=7) 



ERIC 



7 



J 



Items Recommended by All of Original Group (continued) 



F. ORDER OF ITEMS 

lb. The initial items are applicable to all members of the survey population. 
Id. The initial items are nonthreatening. 
le. The initial items are interesting. 

5. If reference ta made to a previous item, that item appears on the same page or on 
the facing page. 

<• Items with similar content are grouped together; within each content group, items 
with the same response format are presented together. 

la. The initial items art clearly connected to the stated purpose of the survey. 
4. Classirication or demographic information is solicited at the end of the instrument unless needed for 
screening pinposes. 

7b. V hin a topic/content area, the items progress from most familiar to least familiar. 

7c. Within a topic/content area, the items progress from least objectionable to most objectionable. 

G. ITEM FORMAT 

8. Response options are arranged vertically (or in columns if several consecutive 
items use the same response options). (n=6) 

9. Response options are close to the item stem. (n=7) 
11. There is adequate space for responding. (n=7) 

13. When ranking, the number of items to be ranked is limited (e.g., three best and 
three worst). (n=5) 

10. The space for responding to items is on the same side of the page throughout the instrument. (n=6) 
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On another 18 items, only one of the original (Hiasc One) participants rated the item as 
"sometimes" or "seldom," whUe the rest accorded it the "usually" rating. Comments or special 
circumstances from those original participants are shown in italics to the right of the item. 

Using the same 80 percent agreement, 12 of these items were supported by the validation 
sample and are again shown in bold-face type. 



Items Usually Recommended by All But One of Original Group 



Item 



Circumstances/Comments 



A3. Margins arc adequate; instrument doesn't 
look crowded. 

A 5. Printing does not bleed through the paper. 

A8. There arc not loo many variations in size and style of type. 

B5. The tone of the directions is polite (e.g., "please"). 

CI. The respondent is able to provide answers to the 
questions in the instrument. 



C4c. For items used for skiiVfilter/screen puqx>scs, instructions 
appear immediately after the response options. (n=6) 

Dlb. Response options are mutually exclusive. 



E3b. Items are simple^ direct* and unambiguous. 

They diLJifit contain instances of loaded*" items 
(that use emotionally colored words). (n=6) 

E3c. Items arc simple, direct, and unambiguous. Thev do not 

contain instances of assumption of an existing state of affairs 
(e.g., "Do you still..."). 

E3g. Items are simple* direct* and unambiguous. They 
do not contain instances of giveaway** words 
(e.gM -air). 

Flc. The initial items are easy. (n=6) 



F7d. Within a topic/content area* the items progress 
from objective to subjective. 



Crowding preferable to longer 
instrument. 



Interesting, yaried format may add 
appeal for children, teens, or 
specialized audiences. 



Person should have the knowledge; 
sometimes opinions of naive 
persons sought (although ibSX dont 
feel able to provide the answers). 



Except items with"check all 
that apply." 

Such words are appropriate in 
attitude surveys. 



NOTE: Wording of this item 
was confusing to some participants. 



NOTE: Wording of this item 
was confusing to some participants. 



More important that initial items 
be interesting if there are no items 
that are both interesting and easy. 

No clear agreement on this. 
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Items Usually Recommended by All But One of Original Group (continued) 



Item 



Cireumstanccs/Commcnts 



FS. Items that require recall are organized by logical 
time sequence. (d=6) 

G2. If necessary, cither sublettering (e.g., 4a, 4b, 4c) or numbering 
by sections (i.e., starting each section with item 1) is used to 
limit the apparent number of items. 

G3. Each item and its response options are on 
the same page. 

G4. Statements or questions, rather than phrases, are used in 
collecting demographic information (e.g., "How old were 
you on your last birthday?" instead of '*Age.'*). (n=6) 

G14b. For check lists, column headings are carried over 
from one page to another. (tt=6) 

G 14c. For check lists, column headings are presented paralld, rather 
than perpendicular, to the item stem. (n^S) 



No clecu' agreement on this. 



Long items (25 Ukert items) 
may not fit on a page. 

Level of detail needed and 
literacy of reader must be 
considered. 

Better not to have to carry over 
to second page. 

There may be too many options 
at times. 
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For the fdlowing 12 items, all but two of the Phase Oae participants agreed that they should be 
recommeiKied Only one of the items, G7, was validated at the 80 percent level 



Items Usually Recommended by All But Two of Original Group 



Item 



Circumstances/Comments 



A7. Size and style of type used for headings is consistent 

throtighout the instnunent Consistency is also evident for 
items and response options. 

A9b. The front page (or cover) contains general diitcuons. 

B4. Instructions are visually different from the body of the 
instrument (e.g., in size and/or style of type). 

Die. Response options include both sides of issue or question. 



E2. 

E3f. 
E3i. 



Bodi sides of issue ( jr neither side) are included in the 
item stem. 



Items are simple, direct aiJ unambiguous. They do not 
contain instances of qualifying clauses, especially at end of stem. 

Ituns are simple, direct, and unambiguous. They do not 
contain mitances of vague terminology (e.g., "the country," 
"just," "fair," "you") 



Within a topic/nontent area, the items progress from 
general to specific. 



Items are numbered with Arabic numerals, (n sS ) 

If an item stem requires two or nrKm lines, the second 
and subsequent lines are indented. (n=S) 

When response options are provided (including, 
if appropriate, a response option of "other"), each 
response option has either a numeric or alphabetic 
code beside it. (n^S) 



G12. Open-ended items are used sparingly. (n^S) 



F7a. 

Gl. 
05. 

G7. 



Preferably in cover letter. 



Both or Mith^r: some issues may 
have more than two viewpoints 

There may be more than two sides. 
Some items may focus on one side 
of an issue. 

NOTE. Wording of this item was 
confusing to some participants. 

Many of the terms acceptable for 
response categories. This statement 
doesn 'r spec^ what part of the 
item the words are to be omitted 
from. 

Sometimes may not care if earlier 
responses influence summary item. 
No clear agreement. 



Recommended to facilitate com- 
puter data entry. Otherwise 
checking beside or circling 
response may be acceptable. 

If most likely responses are known. 
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The foUowing items appear to be more controversial or highly situational in nature than 
conmionly accepted, or the relevant items in this smdy were not clearly written. These items 
would not be included anmng a generic list of questionnaire characteristics. Some of the 
circumstances identified by the respondents are listed. 



Non-Generic, or Situation-Specific Items 



Item 



Circumstances/Conmients 



A4. Paper is white or light-colored with dark ink. 



A9c. The front page (or cover) contains the name of the sponsor. 



A9d The front page (or cover) contains the address of the sponsor. 



A 10. For a multi-page questionnaire, the back page does not 
contain items but may be used for comments. 



Bl. General instnKtions that apply to the entire insmiment 
are provided at the beginning of the instnimeni 

B3c. Instnictions are clear. They provide guidance for expected 
length of open-ended responses. 

C4a. For items used for skip/fllter/screen purposes, the use of this 
type is justified. 



C4d. For items used for skip/filter/screen purposes items pertaining 
to only some of the respondents are indented beneath the 
filler question. 

Die. Response options include a ''don't kiww** option. 



Brighter colors, variedfbrmats may 
be more appealing to children, 
teens, spec^ audiences. 

Some prefer this in cover letter. 
Response rate may be lower if very 
personallcof^idefUial information 
sought. 

Some prefer this in cover letter or 
at end of questionnaire. 
Complete address not needed in 
organiiation survey. 

Use for questions needed rather 

than adding pages. 

Less important if comments have 

been sought throughout the 

questionnaire. 



If there is a need to limit length. 
Available space is an indicator. 

One justification is to shorten the 
overall questionnaire. 
Avoid when possible in mail 
surveys. 

This format may be less appro- 
priate for map-like or flowchart- 
basedformats. 

Some prefer to force respondents 
to makt a choice. 




Non*Generic, or Situation-Specific Items (continued) 



J2n2_ Circumstances/Comments 

D2t, Iiems with Likert*type response options have an appropriately Some prefer not to include a 
labeled midpoint midpoint or to label only the 

end points. 



D3. Sensitive information (e.g., age, salary ) is collected using 
ranges for response options 



E3a. Items are simple, direct, and unambiguous. They do not 
contain instances of jargon, technical terms, or uncommon 
abbreviations. 



E3h. Items are simple, direct, and unambiguous. They do not 
contain instances of inexact words or phrases (e.g., "any," 
"most," "several," "usually," "often," "regularly," "much the 
same"). 

E3j. Items are simple, direct, and unambiguous. They do not 

contain instances of the word "questionnaire" or "check list" 
in heading or text 

F2. If there are any sensitive or difTicult items, they appear in 
the middle or near the end of the instrument, but not at the 
very end 



F3. Open-ended items appear last 



G6. The respondent is asked to circ' ^ or underline responses. 



G14a. For check lists, if long, a line is skipped after every three to 
six items. 



Unless interval level data needed. 
Ranges may be preferable 
anonymity or confidentiality is 
a concern. 

Technical terms can be used \f those 
in the sample would be famiUar 
with them (engineering terms for 
a survey of engineers). 

These are accqxable response 
categories, and this item doesn't 
state whether these words are to be 
omitted from the stem or response. 



All items may be sensitive. 
Place at end unless critical to 
study (more commitment to 
answer because of time already 
spent on the questionnaire). 

Should follow items if used to 
clarify or expand responses to them. 

Except when listing responses 
might influence respondents 
or when possible responses 
cannot be predicted. 

May not be needed if items go 
across most of the page or if 
there is si^cient space between 
items. 



There was much less agrccnicnt on the use of items in section G than in other sections. There 
was also more reluctance to rate the items in that section, possibly indicating confusion over the 
items themselves. 
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Summary 

There were 34 items on which all (or all who rated the item) of Jie Phase One experts agnxd 
that they would usually recommend. For 25 of those items, there was 80 percent or higher support 
fiom the validation panel that the items should be irdudcd in a list of recommendations usually 
made in mail surveys. Of the 18 items on which all but one or the Phase One panel supported, 12 
of the validation panel provided support. And on the 12 items on which all but two of the Hiase 
One panel would usually recommend, only one of the items was supported by the validation panel 
In summary, of the 64 items which a majority of the Phase One participants would usually 
recommend, 38 of them were supported by the validation group at the 80 percent or higher level 
indicating they also would usually make the recommendation. As consensus declined within the 
Phase One participants, the support of the validation group also declined 

There are 18 items from the instrument that appear to be recommendations that would be made 
oiily in certain circumstances or were poorly written and confusing to participants. In a few cases, 
participants indicated they would be more likely not to make such a recommendation than to make 
it. 

The method of analysis for these data was arbitrary. The level-of-agreement criterion for the 
Phase One participants (i.e., all but two or fewer agreeing the reconmiendation would usually be 
made) and the 80 percent criterion for the validation panel may be too lenient The comments from 
the original sample regarding items that are situation-specific or confusing in themselves may lead 
to improvement of the wording or intent of some items and the acceptance that there are conditions 
under which others are applicable. 

Ji:t;;ched "Check List of Desirable Characteristics of Mail Questionnaires" is a compilation 
of those items that were "usually recommended" by at le^t 87.5 percent of the Phase One experts 
(7 of 8) and at least 80 percent of the validation panel. Note that the nature of nx)st of these 
generally-agreed-upon characteristics is very general. Based on this study, it appears that while 
there are sonK mail questionnaire recommendations that can usually be made with some degree of 
confidence, there are other aspects of questionnaire design that are less commonly accepted, and 
their proper use may depend on the experience and knowledge of the researcher regarding not only 
questionnaire dt;sign but also the population to be surveyed, the information sought, and the 
circumstances. In other words, questionnaire design may be a science only up to a certain point; 
beyond that point it is an art, and it would appear that poini is reached somewhere prior to the 
completion of the questionnaire design. 
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I Checfclist of Dtsirable Characteristics of Mail Questionnaires 

GENERAL APPEARANCE 

• The tkle of the sttdy/questioiinaife is likely lo wppe^l lo the survey population. 

• Inamiroem looks easy to oompiete. 

• Magins are adequate; nistrumem(k)esn^k)^ 

• Mating does noiUeed through the paper. 

• Type is clear and legible. 

• Appredatkm for completing the instrument is expitsaed. 

INSTRUCTIONS 

• Instructions are brief. 

• bstnictions are dear 

O lliey specify when to put a check mark and when 10 write in a lespo^ 
O Hiey indicate whether multiple responses aieaUowed. 

• The tone of the directions is polite (e.g., "pl<nse"). 

CHOICE OF ITEMS 

• Hie lespondent is able to provide answers to the questions in the instrument 

• Each item seeks just one piece of infonnation. 

• All items are essential and relevant to tiie purposes of the survey. 

• For items used for skifVfilter/Kreen purposes, instructions are few and simple. 

CHOICE OF RESPONSE OPTIONS 

• ReqxNise options: 

O Exhaust aU possibilities or include "others "undecided," or "neutnti"c^ 
O Are mutually exclusive. 

O Do not contain inore than one alteniative that could be correct unless multipk responses 
aUowed. 

O Are appropriate for tiie item. 

WORDING 

• The choice of words is appropriate to the literKry level of the survey population. 

• Items are simple, direct, and unambiguous. They do not contain instances of: 
O Toaded" items (diat use emotionally cokved words). 

O Double n^atives in items and/dr response options. 

O N^atively worded items coupled with ag^disagree response for^ 

O "Giveaway" words (c.g.. "all"). 

ORDER OF ITEMS 

• The initial items are: 

O Applicable to all members of the survey population. 
O Easy. 

O Nomhreatening. 
O Interesting. 

• If reference is made to a previous item, diat item appears on the same page or on 
the focing page. 

• Items wid) similar content are grouped togedier, widun each content group, items 
witi) the same response format are preaenied togeUier. 

• Witiiin a topic/content area, the items progress from objective to subjective. 

• Items diat require recall are organized by fogkal time sequence. 

ITEM FORMAT 

• Each item and its response options are on die same page. 

• Response options are arranged vertically (or in columns if several consecutive 
items use die same response options). 

• Response options are cfose to die item stem. 

• There is adequate space for responding. 

• When nuiking, die number of items to be ranked is limited (e.g., Uuee best and 
duee worst). 

• For checklists, column headings are carried over from one page to anotiier. 

?9 
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10. 

II. 



Ite Mi tr *i MiMMioMiin ia Hktly to aniad 

USUALLY SOMEHMES 

USUALLY SOMBTUIBS 

ilkiokooiMML ..... USUALLY SOMBUMES 

ii Willi «il||ft|.aMwi*Mtak. USUALLY SOMETIMES 

iM Mt IM teMch pipir. USUALLY SOMETIMES 

1>pt ii aai kiiMi. USUALLY SOMETIMES 

"f*^- USUALLY SOMETIMES 

'^■"■****^''****«^»"«*«yte«f>y!»^ .... USUALLY SOMETTMES 
^ *«Mp^p(cre»««)aMtfM: 

^ ^ I fwmi-aiay dia|iUy«d USUALLY SOMETTMES 

. gwanl Aficilow USUALLY SOMETTMES 

^ ** USUAIXY SOMETTMES 

* ^•^^^^m^ USUALLY SOMETTMES 

i toHmwiy biMt^fareoiwmeali USUALLY SOMETIMES 

Afi^Mw for mill fliilRi die imtnmt ii expittaH USUALLY SOMETIMES 



SELDOM 
SELDOM 
SELDOM 
SELDOM 
SELDOM 
SELDOM 

SELDOM 
sa.DOM 

SELDOM 
SELDOM 
SELDOM 
SELDOM 

SELDOM 
SELDOM 



1. Itm.«««b«dwiAA«b.c«««|. i;sUALLY SOMETTMES SELDOM 

!!?t"^ ^ (1.©., siMii^ tmk Hctfoa 

Willi iimi ]) ii iMd to Hnrit dM ^ipmi amter 

USUALLY SOMETIMES SaOOM 

3. ^i^mdiUfmpmmof^n^o^fi^uHm^ USUALLY SOMEHMES SBJIOM 

^cdl«al«g dwwgn|*fc irf(»^ 

•«yo.«i,«rl.«bi«Miyri«toidofAf-) USUALLY SOMETTMES SELDOM 

^ "^"^^ cwoor mow iki ieoo«d 

.»d«*.«|iilHii.««U-li.,»A USUALLY SOMETTMES SELDOM 

*»«hwA«wiili*«nii«ihto«t USUALLY SOMETTMES SELDOM 



^' ^* ^^ T* toM«|«wMi<(iMiiiliw,ifw 

«ith«.««ricor.lpW»lico«libiiMilt USUALLY SOME11MBS SELDOM 



lf«myo..ic«l^iie«i«id»«„„^,^, USUALLY SOMETTMES SELDOM 



R~po«iopdo«.,,clo«tothili«.ie„, USUALLY SOMETTMES SELDOM 

10. TJ»^ft»iito|«dligtoilwko«<«i««ride 

oril.epH.toigho..«fcita.^t USUALLY SOMETTMES SELDOM 

" Tfci«i,.d.,.-eip,e.for«po«ll.| USUALLY SOMEITMBS SELDOM 

12 Ope««hd|tomi«^^^ ^^^^ SOMETTMES SELDOM 

13. WWifii*ii«.d»MnteoriimtobifM|«|i. 

H"-l«l(M.U«el«,.^toiw«), USUALLY SOMETIMES SELDOM 

14. ForctecUiito: 

^ Vlo^i||Miiikip|MiiAere««yimto 

USUALLY SOMETTMES SELDOM 



USUALLY SOMETTMES SELDOM 



ihM prniMMlitd.,. lo ihe itofn itom. USUALLY SOMETTMES SOJXM 

(VliiM OQMtoM to Hit f) 



:?3 




3. 
4. 



ntiuIlM 
a. cM 

c. Mtjr. 



■McM to tti iMed puqKMe of iIb Mmy.., 
itoaBmaibuBorttBiwvey papylaboii.... 



t«y 



fmikmisiQlkiiBdaiihe 
MfortciwahiipvpoMg. , 




ilww proirew from: 

10 iptdflc 

mm tmmm to kui r«nli« 

«. lMtt nt»cti o«rt ito tonwrtofcjcctkit^ 

A pHtdivt to lab^^iivt 



USUALLY SOMEITMES 

USUALLY SOMETIMES 

USUALLY SOMKHMES 

USUALLY SOMfiUMES 



SELDOM 
SELDOM 
SELDOM 
SELDOM 

USUALLY SOMEHMES SELDOM 



USUALLY SOMEHMES SELDOM 

USUALLY SOMEITMES SELDOM 

USUALLY SOMEHMES SELDOM 

USUALLY SOMETIMES SELDOM 

USUALLY SOMETIMES SELDOM 

USUALLY SOMEITMES SELDOM 

USUALLY SOMETIMES SELDOM 

USUALLY SOMETIMES SELDOM 

USUALLY SOMEITMES SELDOM 

USUALLY SOMETIMES SELDOM 
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providad ti the bminmng of cbe iofiranieiiL 

Instnictiofis arc hnef. 

Iminicibm aedetf: 

a. TVyipiwiywfceatopgi.diBckn^Mrfwheiito 
wnte in • letponiw 

ended tetpofiMi . . 

Initracikm are viMdty (kfTcfeni fixm iie ^ 
intmimeni (tt., m tize uidAsr style of type) 

Tlie tone of the dlractiafu it polile (e.f., "pleaae") 

of the peie, an indication 



«Wei of the pv, an indication 
»•< toe inatnimenl condnoea on the levene tide 
pleaie tarn o»ef") 



II given 

(e J , ^pl. 



USUALLY SOMETTMES SELDOM 

USUALLY SOMETTMES SELDOM 

USUALLY SOMETTMES SELDOM 

USUALLY SOMETIMES SELDOM 

USUALLY SOMETTMES SELDOM 

USUALLY SOMETTMES SELDOM 

USUALLY SOMETIMES SELDOM 

USUALLY SOMETIMES SELDOM 
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Clmirp of Iff 

Tlie ictpofideni it able lo provide enawen lo the 
qiieatiofu in the laitnunenL 

Each item aeekijiMl one pte<e of infoniMCion. ... 

All iiBim aR eaeMlal md lekvani die pwpnej 
of the tnrvcy 

Fnr itetm need for Bkip/filiBi/KieeQ KMvpoaea 

a. The oae of this type it jaeiilied 

b. intiraclioM are few Md tingle. 

c luiruciani appear immedUiBlyaflBr the 

rvtpoate optiont 

d llm pendaini tooidy tome of theretpondenti i 
indented benetfh the niler queation 



USUALLY SOMETIMES SELDOM 

USUALLY SOMETIMES SELDOM 

USUALLY SOMETTMES SELDOM 

USUALLY SOMETTMES SELDOM 

USUALLY SOMETTMES SELDOM 

USUALLY SOMET MBS SELDOM 

USUALLY SOMETTMES SELDOM 
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USUALLY SOMETIMES SELDOM 

USUALLY SOMBITMBS SELDOM 

WUALLY SOMBUMES SELDOM 

USUALLY SOMEHMBS SELDOM 

USUALLY SOMEHMBS SELDOM 

• USUALLY SOMBIIMES SELDOM 

. ... USUALLY SOMEHMBS SELDOM 

USUALLY SOMETIMBS SELDOM 

USUALLY SOMETIMES SELDOM 



3. 




(M^ ^ alvy) ii eollMd Mill 

USUALLY SOMETIMES SELDOM 




SELDOM 




oftM^m^ypoH-lo. USUALLY SOMBUMES SELDOM 



Bom Ml oft 

iietn Hem.... 



a. 

b. 



d 

e. 

f. 
I 



«ct, and MiiMgmi. Thn ^m^ 



Micni.... 



USUALLY SOMEHMBS SELDOM 



USUALLY SOMETIMES SOJXMf 



nyooimwoirii). USUALLY somimms mjxm 



■HMi^NiiM of an ukthi^ sMi of 
(e-I . "Do yom rtll...") 



rupoM* rorawL 

q«'i^etoii«.eip«i.Uyiiendoritent 

'fiveaway- wo«dt («,|., -aU'y. 

•■•Uy. oftM, R|«larty.- -nmdi Ae aami-y.. . 
v^ae tomiaolqur (e-g., 'iIm oowMry.- 'jm,' "fair." 

■yoi") * 

*e wold "qaMiaimiira"or "cMMm" k Maa 
Of text ^ 



USUALLY SOimMBS SELDOM 

USUALLY SOimiMBS SELDOM 

USUALLY SOMETIMES SELDOM 

USUALLY SOMETIMES SODOM 

USUALLY SOMEHMES SELDOM 

USUAUY SOMETIMES SELDOM 

USUAU.Y SOMETIMBS SELDOM 

USUAUY SOMBTTMBS SELDOM 
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